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BRIEF OF APPELLANT 
The Appellant has filed a timely Notice of Appeal from the action of the 
Examiner in finally rejecting all of the claims that were considered in this 
appUcation. This Brief is being filed under the provisions of 37 C.F.R. § 1.192. 
The Filing Fee, as set forth in 37 C.F.R. § 1.17(c), is submitted herewith. 



REAL PARTY IN INTEREST 
The real party in interest is Microsoft Corporation, the assignee of all right, 
title and interest in and to the subject invention by way of assignment from Rui, 
who is the named inventive entity and is captioned in the present brief, to the 
Microsoft Corporation by way of assignment document recorded at Reel/Frame 
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11421/0403 in the United States Patent and Trademark Office on December 28, 
2000. 

RELATED APPEALS AND INTERFERENCES 
Appellant is not aware of any other appeals or interferences which will 
directly affect, be directly affected by, or otherwise have a bearing on the Board's 
decision on this pending appeal. 

STATUS OF CLAIMS 

Claims 2-8, 10-30, 33, 34, 46-49, and 52-66 stand rejected and are pending 
in this Application. Claims 2-8, 10-30, 33, 34, 46-49, and 52-66 are appealed. 
Claims 5, 6, 8, 19, 27, 28, 33, and 55 were previously amended. Claims 31, 39 
and 45 were previously canceled. Claims 1, 9, 32, 35-38, 40-44, 50 and 51 are 
cancelled in the Amendment being filed herewith. Claims 2-8, 10-30, 33, 34, 46- 
49, and 52-66 are set forth in the Appendix of Appealed Claims. 

Claims 8, 48, 49 and 57 stand rejected under 35 U.S.C. § 102(b) as being 
anticipated by an article entitled "Relevance Feedback Techniques in Interactive 
Content-Based Image Retrieval" by Rui et al. (hereinafter "RFT"). 

Claims 11-18, 46, 47 and 60-66 stand rejected under 35 U.S.C. 102(a) as 
being anticipated by "MindReader: Querying Databases Through Multiple 
Examples" by Ishikawa et al. (Proc. 24th VDLB Conf. (New York), 1988, pp. 
2 1 8-227) (hereinafter "MR"). 

Claims 2-7, 19-30, 33, 34, 53-56, 58 and 59 stand rejected under 35 U.S.C. 
§ 103(a) as being unpatentable over RFT in view of MR. 
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Claim 10 stands rejected under 35 U.S.C. § 103(a) as being unpatentable 
over RFT in view of an article entitled "Water-Filling: A Novel Way For Image 
Structural Feature Extraction" by Zhou et al., Proc. Of IEEE Intemational 
Conference on Image Processing, Kobe, Japan, October 1999 (hereinafter 
"Zhou"). 

Claim 52 stands rejected under 35 U.S.C. § 103(a) as being unpatentable 
over RFT in view of U.S. Patent No. 6,504,571 to Narayanaswami et al 
(hereinafter "Narayanaswami"). 

STATUS OF AMENDMENTS 
A Final Office Action was issued on December 24, 2003. 
Appellant filed a Notice of Appeal on April 12, 2004 in response to the 
Final Office Action. 

An Amendment is being filed concurrently which cancels Claims 1, 9, 32, 
35-38, 40-44, 50 and 51. Claims 2, 7, 10, 33 and 52 are rewritten in independent 
form. Accordingly, the Amendment does not necessitate a new search, does not 
raise the issue of new matter, does not present additional claims without canceling 
a corresponding number of finally rejected claims, and does not otherwise 
introduce new issues. Thus, the amendment complies with the requirements 
specified by MPEP §1207. 

SUMMARY OF INVENTION 
In accordance with one aspect, improved image retrieval based on 
relevance feedback is described herein. A hierarchical (per-feature) approach is 
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used in comparing images. Multiple query vectors are generated for an initial 
image by extracting multiple low-level features from the initial image. When 
determining how closely a particular image in an image collection matches that 
initial image, a distance is calculated between the query vectors and corresponding 
low-level feature vectors extracted from the particular image. Once these 
individual distances are calculated, they are combined to generate an overall 
distance that represents how closely the two images match. 

According to another aspect, when a set of potentially relevant images are 
presented to a user, the user is given the opportunity to provide feedback regarding 
the relevancy of the individual images in the set. This relevancy feedback is then 
used to generate a new set of potentially relevant images for presentation to the 
user. The relevancy feedback is used to influence the generation of the query 
vector, influence the weights assigned to individual distances between query 
vectors and feature vectors when generating an overall distance, and to influence 
the determination of the distances between the query vectors and the feature 
vectors. 

According to another aspect, the calculation of a distance between a query 
vector and a feature vector involves the use of a matrix to weight the individual 
vector elements. The type of matrix used varies dynamically based on the number 
of images for which feedback has been received from the user and the number of 
feature elements in the feature vector. If the number of images for which feedback 
has been received is less than the number of feature elements, then a diagonal 
matrix is used (which assigns weights to the individual vector elements in the 
distance calculation). However, if the number of images for which feedback has 
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been received equals or exceeds the number of feature elements, then a full matrix 
is used (which transforms the low-level features of the query vector and the 
feature vector to a higher level feature space, as well as assigns weights to the 
individual transformed elements in the distance calculation). 
[Excerpts Taken from the Application, Pages 2-4]. 

ISSUES 

1. Whether Claims 8, 48, 49, and 57 were properly rejected under 35 
U.S.C. § 102(b) as being anticipated by RFT. 

2. Whether Claims 11-18, 46-47 and 60-66 were properly rejected 
under 35 U.S.C. § 102(a) as being anticipated by MR. 

3. Whether Claims 2-7, 19-30, 33-34, 53-56 and 58-59 were properly 
rejected under 35 U.S.C. § 103(a) as being unpatentable over RFT in view of MR. 

4. Whether Claim 10 was properly rejected under 35 U.S.C. § 103(a) as 
being unpatentable over RFT in view of Zhou. 

5. Whether Claim 52 was properly rejected under 35 U.S.C. § 103(a) as 
being unpatentable over RFT in view of Narayanaswami. 

GROUPING OF CLAIMS 
All of the claims do not stand or fall together. The claims are grouped as 
follows: 

1. First Ground of Rejection: The ground of the rejection based on 35 
U.S.C. § 102(b) directed toward pending Claims 8, 48, 49, and 57 as being 
anticipated by RFT, these claims stand or fall together as to this rejection. 
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II. Second Ground of Rejection: The ground of the rejection based on 
35 U.S.C. § 102(a) directed toward pending Claims 11-18 and 60-66 as being 
anticipated by MR, these claims stand or fall together as to this rejection. 

III. Third Ground of Rejection: The ground of the rejection based on 35 
U.S.C. § 102(a) directed toward pending Claims 46-47 as being anticipated by 
MR, these claims stand or fall together as to this rejection. These claims are 
separately patentable from the claims of the Second Ground of Rejection and 
therefore do not stand or fall together with the claims of the Second Ground of 
Rejection with respect to MR under 35 U.S.C. § 102(a). 

IV. Fourth Ground of Rejection: The ground of the rejection based on 35 
U.S.C. § 103(a) directed toward pending Claims 2, 3, 5, 6, 19-22, and 53 as being 
unpatentable over RFT in view of MR, these claims stand or fall together as to this 
rejection. 

V. Fifth Ground of Rejection: The ground of the rejection based on 35 
U.S.C. § 103(a) directed toward pending Claims 7, 55, and 56 as being 
unpatentable over RFT in view of MR, these claims stand or fall together as to this 
rejection. These claims are separately patentable from the claims of the Fourth 
and Sixth Grounds of Rejection and therefore do not stand or fall together with the 
claims of the Fourth and Sixth Grounds of Rejection with respect to RFT in view 
of MR under 35 U.S.C. § 103(a). 

VI. Sixth Ground of Rejection: The ground of the rejection based on 35 
U.S.C. § 103(a) directed toward pending Claims 4, 23-30, 33, 34, and 59 as being 
unpatentable over RFT in view of MR, these claims stand or fall together as to this 
rejection. These claims are separately patentable from the claims of the Fourth 
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and Fifth Grounds of Rejection and therefore do not stand or fall together with the 
claims of the Fourth and Fifth Grounds of Rejection with respect to RFT in view 
of MR under 35 U.S.C. §103(a). 

VII. Eighth Ground of Rejection: The ground of the rejection based on 35 
U.S.C. § 103(a) directed toward pending Claim 10 as being unpatentable over 
RFT in view of Zhou. 

VIIL Ninth Ground of Rejection: The ground of the rejection based on 35 
U.S.C. § 103(a) directed toward pending Claims 52 and 54 as being unpatentable 
over RFT in view of Narayanaswami. 

ARGUMENT 

First Ground of Rejection. Claims 8, 48, 49, and 57 satisfy the requirements of 
35 U.S.C. § 102(b) so as to be unanticipated by RFT. 

1. Claims 8, 48, 49, and 57 recite a particular equation for 
calculation of weight between a query vector and a feature vector 

Claims 8 and 57 recite the following: 

• wherein represents a summation, over the 

images in the set of potentially relevant images, 
of a product of a relevance of the image and a 
distance between the query vector and the 
feature vector, and 

• wherein the selecting a new set of potentially 
relevant images comprises combining, for each 
image, a weighted distance between the 
plurality of query vectors and the plurality of 
feature vectors, and 
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• wherein the weight (w.) for each of a plurality 

( / ) of distances between a query vector and a 
corresponding feature vector is calculated as: 




Claim 48 recites "generating a weight (w.) for each of a plurality ( / ) 
of distances between a query vector corresponding to one of a plurality ( / ) 
of features and a feature vector corresponding to the one of the plurality ( / ) 
of features as: w,. = ^ 

Neither RFT, nor any of the other submitted references, alone or in 
combination, disclose, teach or suggest the above limitations and more 
specifically, the equation as follows: 

(hereinafter "particular equation") as recited by Claims 8 and 48. 

2. RFT's calculation of similarity is not a calculation of 
weight as claimed 

The Office asserts RFT, and more particularly steps 4-6 of RFT's 
method, as disclosing the particular equation for calculating weight. 
However, RFT describes an algorithm (Calcs. 3-8) wherein a query object 
Q is described in terms of features F having associated presentation vectors 
rij (Calcs. 3-5 and associated text). The user's information need is 
distributed among different features fj of the query object Q, according to 
their corresponding weights Wy. The various objects* similarities to the 
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query Q is calculated according to a similarity measure my and the weights 
Wijk (Calc. 6). See RFZ Page 4, 

RFT teaches use of linear techniques to allow two levels of 
calculation to be combined into one, as represented in Calc. 9. See RFT, 
Page 4. RFT, however, does not disclose, teach or suggest the particular 
equation as recited by Claims 8, 48, 49, and 57 for calculation of weights. 
The weights calculated as described in Claims 8, 48, 49 and 57 are a 
summation of square roots of summations of products of relevance and 
distance between the query vector and the feature vector. A square root 
operation is not a linear operation, 

3. Claims 8, 48. 49 and 57 are not anticipated by RFT 

In order to provide a valid finding of anticipation, several conditions 
must be met: (i) the reference must include every element of the claim 
within the four comers of the reference {See MPEP §2121); (ii) the 
elements must be set forth as they are recited in the claim (See MPEP 
§2131); (iii) the teachings of the reference cannot be modified {See MPEP 
§706.02, stating that "No question of obviousness is present" in conjunction 
with anticipation); and (iv) the reference must enable the invention as 
recited in the claim {See MPEP §2121.01). Additionally, anticipation 
requires that (v) these conditions must be simultaneously satisfied. See 
MPEP §212L0L Accordingly, the §102(b) rejection of Claims 8, 48, 49 
and 57 is believed to be in error. 
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The PTO and Federal Circuit provide that §102 anticipation requires 
that each and every element of the claimed invention be disclosed in a 
single prior art reference . In re Spada, 911 FJd 705, 15 USPQld 1655 
(Fed. CiK 1990) (emphasis added). The corollary of this rule is that the 
absence from a cited §102 reference of any claimed element negates 
anticipation. Kloster Speedsteel AB, et ah v. Crucible, Inc., et al, 793 F,2d 
1565, 230 USPQ 81 (Fed, Cir, 1986) (emphasis added). 

In this instance, RFT does not disclose, as recited in Claims 8 and 
57, "wherein the weight (w,) for each of a plurality (/) of distances 
between a query vector and a corresponding feature vector is calculated as: 



Also, RFT does not disclose "generating a weight (m.) for each of a 
plurality ( / ) of distances between a query vector corresponding to one of a 
plurality ( / ) of features and a feature vector corresponding to the one of the 



Claim 49 is a dependent claim that depends directly from 
independent Claim 48. Because Claim 49 incorporates all the limitations 
of the Claim 48 from which it depends, this claim is patentable for at least 
the same reasons for which Claim 48 is patentable. 

For at least these reasons, Claims 8, 48, 49, and 57 satisfy the 
requirements of 35 U.S.C. § 102(b) so as to be unanticipated by RFT. The 
Appellant respectfully requests the Board to overtum the First Ground of 
Rejection. 




plurality ( / ) of features as: u- = /— " as recited in Claim 48. 
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Second Ground of Rejection. Claims 1 1-18 and 60-66 satisfy the requirements of 
35 U.S.C. § 102(a) so as to be unanticipated by MR. 

1. Claims 11-18 and 60-66 recite selection of one of two types 
of matrixes 

Independent Claims 11 and 60 each recite selecting between two 
types of matrixes to be used to weight, based on relevance feedback, a 
plurality of feature elements for image retrieval. One of the two types of 
matrixes is selected based on both a number of previously retrieved 
relevant images and a length of a feature vector including the pluraUty of 
feature elements. 

2. MR does not disclose selection of matrixes 

MR discloses use of a symmetric matrix M as a generalized ellipsoid 
distance matrix relating to distances between a query point described by a 
vector q and sample vectors x in a data point matrix X. See MR, Pages 220- 
22 L The Office asserts theorems 2-3 of section 3.4 and theorems 2-3 and 
corresponding appendixes C-D of MR for disclosure of selection. For 
instance, the Office asserts, "the covariance matrix (C) is the basis of the 
selection decision (if C is invertible or singular/non-invertible), but it is not 
one of the types of matrixes "used to weight" as claimed." Office Action 
Dated December 24, 2003, Page 19, The Appellant respectfully disagrees. 
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Appendix D of MR merely describes how the Moore-Penrose 
inverse matrix is calculated. For example, the result of the proof is "matrix 
[which] is called the Moore-Penrose inverse matrix (or pseudo-inverse 
matrix) of C". MR, Page 227. Thus, contrary to the assertions made by the 
Office, the covariance matrix (C) is not utilized as the basis for the 
selection decision, but rather merely shows how to calculate a Moore- 
Penrose inverse matrix. Indeed, nowhere in MR is selection even 
discussed. 

3. Claims 11-18 and 60-66 are not anticipated by MR 

As previously described, the PTO and Federal Circuit provide that 
$102 anticipation requires that each and every element of the claimed 
invention be disclosed in a single prior art reference . See In re Spada^ 911 
FJd 705, 15 USPQ2d 1655 (Fed, Cir, 1990). MR does not disclose 
"selecting one of the two types of matrixes based on both a number of 
previously retrieved relevant images and a length of a feature vector 
including the plurality of feature elements" as recited in Claim 1 1 nor 
"select one of the two types of matrixes based on both a number of 
previously retrieved relevant images and a length of a feature vector 
including the plurality of feature elements" as claimed in Claim 60. 

Although MR discloses a matrix M and a matrix C^, MR does not 
provide any selection criteria for when to discriminate between the 
matrixes. Claims 11 and 60, however, recite selection of matrixes "based 
on both a number of previously retrieved relevant images and a length of a 
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feature vector including the plurality of feature elements" which is not 
disclosed by MR. Contrary to the Office's assertion, disclosure of a setting 
involving "multimedia systems and digital libraries that handle mixed 
media" by MR does not disclose "a number of previously retrieved relevant 
images" as utilized to select a matrix as claimed. See MR, Page 219, 
Column 2, Lines 1-2. 

Claims 12-18 are dependent claims that depend directly or 
indirectly from Independent Claim 11. Claims 61-66 are dependent claims 
that depend directly or indirectly from Independent Claim 60. Because 
Claims 12-18 and 61-66 incorporate all the limitations of the claims they 
respectively depend from, these claims are patentable for at least the same 
reasons for which Claims 1 1 and 60 are patentable. 

For at least these reasons, Claims 11-18 and 60-66 satisfy the 
requirements of 35 U.S.C, § 102(a) so as to be unanticipated by MR. The 
Appellant respectfiiUy requests the Board to overtum the Second Ground of 
Rejection. 

Third Ground of Rejection. Claims 46-47 satisfy the requirements of 35 U.S.C. 
§ 102(a) so as to be unanticipated by MR. These claims are separately patentable 
from the claims of the Second Ground of Rejection with respect to MR due to 
recitation of an equation which is utilized to generate a query vector which is not 
disclosed by MR. Therefore, these claims do not stand or fall together with the 
claims of the Second Ground of Rejection with respect to MR under 35 U.S.C. 
§ 102(a). 



13 



Application No. 09/660,536 



1. Claims 46 and 47 recite generation of a query vector from 
a transposition of a vector 

Claim 46 recites a method of generating a query vector to compare 
to a feature vector of another image. The method includes receiving 
feedback regarding the relevance of each image of a set of images, wherein: 

• N represents the number of images in the set of images 
for which user relevance feedback has been received, 

• TT^ represents the relevance of image n in the set of 
images, 

• TT represents a transposition of a vector generated by 
concatenating the individual tt^ values, and 

• X represents an image matrix that is generated by 
stacking N training vectors corresponding to the set of 
images into a matrix. 

A query vector (^) is generated corresponding to one of a plurality of 
features as follows: 

^ K X 
^=-J^ • 

This generation of the query vector as described by the above equation is 
not disclosed by MR. 

2. MR recites a sample vector formed from transposition of 
a data point matrix 
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The Office asserts Section 3 and specifically Theorem 1 and Table 1 
for the details of generating a query vector to compare to a feature vector of 
another image. See Office Action Dated December 23, 2003, Page 8. The 
Appellant respectfully disagrees. 

MR discloses an average of data vectors which are weighted by 
goodness values. For instance, MR recites with respect to Theorem 1 that a 
query vector = X^(goodness values)/(sum of goodness values), where X 
represents a data point matrix (see Table I). This is not the same as the 
recitation of claim 46. In other words, a transposition of the data point 
matrix multiplied by the goodness values for the examples does not 
disclose a transposition of a vector generated by concatenating the 
individual values multiplied by an image matrix that is generated by 
stacking training vectors corresponding to the set of images into a matrix. 

3. Claims 46 and 47 are not anticipated by MR 

As previously described, the PTO and Federal Circuit provide that 
§102 anticipation requires that each and every element of the claimed 
invention be disclosed in a single prior art reference . See In re Spada, 911 
F.2d 705, 15 USPQ2d 1655 (Fed, Cir 1990) (emphasis added), MR does 
not disclose a query vector as described in Claim 46 which is formed by 
multiplying a transposition of a vector (generated by concatenating the 
individual values that represent the relevance of image n in the set of 
images) by an image matrix (generated by stacking training vectors 
corresponding to the set of images into a matrix), the result of which is then 
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divided by a sum of the relevance of image in the set of images. Rather, 
MR discloses a transposition of a data point matrix multiplied by goodness 
values for examples , the result of which is then divided by a sum of the 
goodness values. 

Claim 47 is a dependent claim that depends directly from 
Independent Claim 46. Because Claim 47 incorporates all the limitations of 
Claim 46, this claim is patentable for at least the same reasons for which 
Claim 46 is patentable. 

For at least these reasons. Claims 46-47 satisfy the requirements of 
35 U.S.C. § 102(a) so as to be unanticipated by MR. The Appellant 
respectfully requests the Board to overturn the Third Ground of Rejection. 

Fourth Ground of Rejection. Claims 2, 3, 5, 6, 19-22 and 53 satisfy the 
requirements of 35 U.S.C. § 103(a) so as to be patentable over RFT in view of MR. 

1. Claims 2, 3. 5, 6, 19-22 and 53 describe dynamic selection 

Independent Claim 2 describes "dynamically selecting the matrix 
based on both a number of images in the set of potentially relevant images 
for which relevance feedback was input and a number of feature elements 
in the one feature vector". Independent Claim 19 describes "dynamically 
selecting a type of matrix to use based on both the user feedback and the 
number of the plurality of feature elements". Neither RFT nor MR, alone 
or in combination, disclose, teach or suggest these aspects. 
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2. Neither RFT nor MR, alone or in combination, disclose 
dynamic selection as claimed 

The Office correctly asserts that "RFT does not expUcitly disclose 
'dynamically selecting the matrix,.,' as claimed". See Office Action Dated 
December 24, 2003, Page 11. The Office then asserts MR to cure the 
defects of RFT. 

As previously described in relation to the Second Ground of 
Rejection, which is hereby incorporated into this rejection, MR fails to 
disclose selection of matrixes as claimed. Rather, MR discloses use of a 
symmetric matrix M as a generalized ellipsoid distance matrix relating to 
distances between a query point described by a vector and sample vectors 
in a data point matrix. See MR, Pages 220-22 L 

MR provides no disclosure of "dynamically selecting the matrix 
based on both a number of images in the set of potentially relevant images 
for which relevance feedback was input and a number of feature elements 
in the one feature vector" as claimed in Claim 2. Neither does MR provide 
disclosure for "dynamically selecting a type of matrix to use based on both 
the user feedback and the number of the plurality of feature elements" as 
claimed in Claim 19. Accordingly, neither RFT nor MR, alone or in 
combination, disclose, teach or suggest dynamic selection as claimed. 

3. Claims 2, 3, 5, 6, 19-22, and 53 are nonobvious over RFT 
in view of MR 
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To establish prima facie obviousness of a claimed invention, all the 
claim limitations must be taught or suggested by the prior art. In re Ryoka, 
180 UXP.Q, 580 (CCPA. 1974), See also In re Wilson, 165 U.S.P.Q, 
494 (CCPA, 1970). Neither RFT nor MR, alone or in combination, 
disclose teach or suggest dynamic selection as claimed nor selection criteria 
to be used in dynamic selection as claimed. 

Dynamic selection 

As correctly stated by the Office, RFT does not disclose 
"dynamically selecting the matrix" or "dynamically selecting a type of 
matrix" as claimed in Claims 2 and 19, respectively. Indeed, RFT does not 
even mention a matrix, and therefore cannot teach or suggest selection of a 
matrix. 

MR does not cure the defects of RFT. As previously described in 
relation to the Second Ground of Rejection, MR fails to disclose, teach or 
suggest any type of selection. Although MR does mention a second matrix, 
MR merely describes how the matrix is calculated. For example, the result 
of the proof of Theorem 3 is "matrix [v^hich] is called the Moore- 
Penrose inverse matrix (or pseudo-inverse matrix) of C". MR, Page 227. 
Therefore, neither RFT nor MR, alone or in combination, disclose teach or 
suggest dynamic selection as claimed. 

Selection Criteria 

Additionally, neither RFT nor MR, alone or in combination, 
disclose, teach or suggest selection criteria to be used for dynamic selection 
as claimed. Claim 2 recites dynamic selection of matrixes " based on both a 
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number of images in the set of potentially relevant images for which 
relevance feedback was input and a number of feature elements in the one 
feature vector". Claim 19 recites "dynamically selecting a type of matrix to 
use based on both the user feedback and the number of the plurality of 
feature elements ". 

As previously stated, RFT does not even disclose a matrix. 
Therefore, RFT does not and cannot disclose selection criteria used for 
dynamic selection of matrixes as claimed in Claims 2 and 19. 

MR does not cure the defects of RFT. MR merely describes the 
existence of different matrixes, but does not teach or suggest selection 
between the matrixes nor teach or suggest criteria for how such selection 
would be made. Therefore, neither RFT nor MR, alone or in combination, 
disclose teach or suggest selection criteria as claimed. 

Claims 3, 5, and 6 are dependent claims that depend directly or 
indirectly from Independent Claim 2. Claims 20-22 and 53 are dependent 
claims that depend directly or indirectly from Independent Claim 19. 
Because Claims 3, 5, 6, 20-22 and 53 incorporate all the limitations of the 
claims they respectively depend from, these claims are patentable for at 
least the same reasons for which Claims 2 and 19 are patentable. 

Accordingly, Claims 2, 3, 5, 6, 19-22, and 53 satisfy the 
requirements of 35 U.S.C. § 103(a) so as to be patentable over RFT in view 
of MR. The Appellant respectfully requests the Board to overturn the 
Fourth Ground of Rejection. 
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Fifth Ground of Rejection. Claims 7, 55, and 56 satisfy the requirements of 35 
U.S.C. § 103(a) so as to be patentable over RFT in view of MR. These claims are 
separately patentable from the claims of the Fourth and Sixth Grounds of 
Rejection with respect to RTF in view of MR due to recitation of an equation 
which is utilized to generate a query vector which is not disclosed by RTF in view 
of MR, alone or in combination. Therefore, these claims do not stand or fall 
together with the claims of the Fourth and Sixth Grounds of Rejection with respect 
to RTF in view of MR under 35 U.S.C. § 103(a). 



1. Claims 7^ 55 and 56 recite generation of a query vector 
from a transposition of a vector 

Claims 7 and 55 recite generation of a query vectors (g) using the 
following equation: 

q = , wherem: 

• represents the number of images in the set of images 
for which user relevance feedback has been received, 

• TT^ represents the relevance of image n in the set of 
images, 

• TT represents a transposition of a vector generated by 
concatenating the individual tt^ values, and 

• X represents an image matrix that is generated by 
stacking N training vectors corresponding to the set of 
images into a matrix. 
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Neither RFT nor MR, alone or in combination, disclose, teach or suggest 
these aspects. 

2. RFT does not teach or suggest generation of a new query 
vector, and MR does not cure the defects of RFT 

The Office correctly asserts that "RFT does not explicitly disclose 
the details of the generation of a new query vector as claimed because RFT 
is silent on these details". See Office Action Dated December 24, 2003, 
Page 13. The Office then incorrectly asserts MR to cure the defects of 
RFT. 

As previously described in relation to the Third Ground of Rejection, 
MR discloses an average of data vectors which are weighted by goodness 
values. For instance, MR recites with respect to Theorem 1 that a query 
vector = X (goodness values)/(sum of goodness values), where X 
represents a data point matrix (see Table I). This is not the same as the 
recitation of claim 46, In other words, a transposition of the data point 
matrix multiplied by the goodness values for the examples does not 
disclose a transposition of a vector generated by concatenating the 
individual values multiplied by an image matrix that is generated by 
stacking training vectors corresponding to the set of images into a matrix. 
Neither RFT nor MR nor any of the other submitted references, alone or in 
combination, cure these defects. 
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3. Claims 7, 55, and 56 are nonobvious over RFT in view of 

MR 

To establish prima facie obviousness of a claimed invention, all the 
claim limitations must be taught or suggested by the prior art. In re Ryoka, 
180 US,P,Q. 580 (CCPA, 1974). See also In re Wilson, 165 US.P.Q. 
494(CCP.A. 1970). 

Neither RFT, nor MR, alone or in combination, disclose teach or 
suggest a query vector as described in Claims 7 and 55 which is formed by 
multiplying a transposition of a vector (generated by concatenating the 
individual values that represent the relevance of image n in the set of 
images) by an image matrix (generated by stacking training vectors 
corresponding to the set of images into a matrix), and then divided by a 
sum of the relevance of image in the set of images. 

Claim 56 is a dependent claim that depends directly from 
Independent Claim 55. Because Claim 56 incorporates all the limitations of 
Claim 55, this claim is patentable for at least the same reasons for which 
Claim 55 is patentable. 

For at least these reasons, Claims 7, 55 and 56 satisfy the 
requirements of 35 U.S.C. § 103(a) so as to be nonobvious over RFT in 
view of MR. The Appellant respectfully requests the Board to overturn the 
Fifth Ground of Rejection. 

Sixth Ground of Rejection. Claims 4, 23-30, 33, 34, and 59 satisfy the 
requirements of 35 U.S.C. §103(a) so as to be patentable over RFT in view of MR. 
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These claims are separately patentable from the claims of the Fourth and Fifth 
Grounds of Rejection with respect to RTF in view of MR due to recitation of using 
one matrix that transforms a query vector and one feature vector to a higher-level 
feature space and then using another matrix that assigns a weight to each element 
of the transformed query vector and the transformed feature vector, which is not 
taught or suggested by RTF in view of MR, alone or in combination. Therefore, 
these claims do not stand or fall together with the claims of the Fourth and Fifth 
Grounds of Rejection with respect to RTF in view of MR under 35 U.S.C. 
§ 103(a). 

1. Claims 4, 23-30, 33, 34. and 59 recite transformation of 
vectors to a higher-level feature space and then assigning a weight 

Claim 4 recites wherein the dynamically selecting comprises: 

• if the number of images in the set of potentially relevant images 
for which relevance feedback was input is not less than the 
number of feature elements in the one feature vector, then using 
one matrix that transforms the query vector and the one feature 
vector to a higher-level feature space and then using another 
matrix that assigns a weight to each element of the transformed 
query vector and the transformed feature vector; and 

• if the number of images in the set of potentially relevant images 
is less than the number of feature elements in the one feature 
vector, then using a matrix that assigns a weight to each element 
of the query vector and the one feature vector 
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Claims 23 and 30 recite the following: 

• if the number of training samples either equals or exceeds a 
threshold amount, then determining a distance between the query 
vector and the feature vector including transforming the query 
vector and the feature vector to a higher-level feature space and 
then assigning a weight to each element of the transformed query 
vector and the transformed feature vector; and 

• if the number of training samples does not exceed the threshold 
amount, then determining the distance between the query vector 
and the feature vector including assigning a weight to each 
element of the query vector and the feature vector. 

Claim 33 recites "mapping the low-level feature vector to a higher 
level feature space". 

Claim 59 recites use of "one matrix that transforms the query vector 
and the one feature vector to a higher-level feature space and then using 
another matrix that assigns a weight to each element of the transformed 
query vector and the transformed feature vector when the number of images 
in the set of potentially relevant images for which relevance feedback was 
input is not less than the number of feature elements in the one feature 
vector". 

Neither RFT nor MR, alone or in combination, disclose, teach or 
suggest these aspects as described in Claims 4, 23-30, 33, 34, and 59. 

Beginning at page 15 of the subject application, exemplary 
comparison of a query vector and a corresponding feature vector is 
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described. For example, to compare the query vector (q.) and a 

-> 

corresponding feature vector of an image m (x^- ), the distance between the 
two vectors is determined. The calculation to determine the distance g^^ 
can be written as: 

^..=(^(^-C))'A,(/>(^-xl.)) 

where the low-level feature space is transformed into the higher level 
feature space by the mapping matrix P. and then weights are assigned to the 
feature elements of the new feature space by the weighting matrix A ■ . 

2. Neither RFT nor MR, alone or in combination, disclose, 
teach or suggest transformation of vectors to a higher-level feature 
space and then assigning a weight 

The Office merely relies on RFT for disclosure of a system as 
described in relation to the rejection of Claim 2. Accordingly, the 
arguments made in relation to the Second Ground of Rejection are hereby 
incorporated with respect to the rejection of the present claims. 

The Office then asserts MR to correct the defects of RFT, namely to 
teach dynamic selection. The Appellant respectfully disagrees. As 
previously described in relation to the Second and Fourth Grounds of 
Rejection, the contents of which are hereby incorporated into this ground of 
rejection, MR does not disclose dynamic selection. MR merely discloses 
use of a symmetric matrix M as a generalized ellipsoid distance matrix 
relating to distances between a query point described by a vector q and 
sample vectors x in a data point matrix X. See MR, Pages 220-22 L 
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In particular, the Office asserts Appendix D of MR for disclosure of 
dynamic selection through use of two matrixes as claimed. Appendix D 
merely describes a Moore Penrose inverse matrix. There is no description 
in MR or RFT for " transforming the query vector and the feature vector to a 
higher-level feature space", and " then assigning a v^eight to each element of 
the transformed query vector and the transformed feature vector", as recited 
in claims 4, 23 and 30. (emphasis added). 

3. Claims 4, 23-30. 33, 34 and 59 are nonobvious over RFT 
in view of MR 

To establish prima facie obviousness of a claimed invention, all the 
claim limitations must be taught or suggested by the prior art. In re Ryoka, 
180 UXP.Q. 580 (CCPA. 1974), See also In re Wilson, 165 UXP.Q. 
494 (cap A. 1970). Neither RFT nor MR, alone or in combination, 
disclose teach or suggest transformation of vectors to a higher-level feature 
space and then assigning a weight as claimed in Claims 4, 23, 30, and 59. 
Additionally, neither RFT nor MR, alone or in combination, disclose, teach, 
or suggest "mapping the low-level feature vector to a higher level feature 
space" as claimed in Claim 33. 

Additionally, Claims 24-29 are dependent claims that depend 
directly or indirectly from Independent Claim 23. Claim 34 is a dependent 
claim that depends directly from Claim 33. Because Claims 24-29 and 34 
incorporates all the limitations from respective Claims 23 and 33, these 
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claims are patentable for at least the same reasons for which Claims 23 and 
33 are patentable. 

For at least these reasons, Claims 4, 23-30, 33, 34 and 59 satisfy the 
requirements of 35 U.S.C. § 103(a) so as to be nonobvious over RFT in 
view of MR. The Appellant respectfully requests the Board to overturn the 
Sixth Ground of Rejection. 

Seventh Ground of Rejection. Claim 10 satisfies the requirements of 35 U.S.C. 
§ 103(a) so as to be patentable over RFT in view of Zhou. 

Zhou is not available as a Reference 

The Office asserts RFT in view of Zhou. Zhou, however, is not 
available as a reference. Zhou has a reference date of October, 1999. See 
Water-Filling: A Novel Way for Image Structural Feature Extraction, by 
Xiang Sean Zhou, Yong Rui, and Thomas S. Huang, Proc, of IEEE 
Intemational Conference on Image Processing, Kobe, Japan, October 1999. 
The subject application, however, claims the benefit of U.S. Provisional 
Apphcation No. 60/153,730, filed September 13, 1999. Therefore, it is 
respectfully submitted that the Office has not made a prima facie rejection 
for obviousness because Zhou is not available as a reference. Accordingly, 
Claim 10 satisfies the requirements of 35 U.S.C. § 103(a) so as to be 
patentable over RFT in view of Zhou. The Appellant respectfully requests 
the Board to overturn the Seventh Ground of Rejection. 
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Eighth Ground of Rejection. Claims 52 and 54 satisfy the requirements of 35 
U.S.C. § 103(a) so as to be patentable over RFT in view of Narayanaswami. 



1. Claims 52 and 54 recite feedback via speech recognition 

Claims 52 and 54 recite "receiving feedback comprises receiving 
feedback via speech recognition" which is not disclosed, taught or 
suggested by any of the submitted references, alone or in combination. 



2. RFT does not disclose Speech Recognition, and 
Narayanaswami does not cure the defects of RFT 

The Office correctly asserts that RFT does not explicitly disclose 
that the receiving comprises receiving feedback via speech recognition as 
claimed. See Office Action Dated December 24, 2003, Page 10. The 
Office then incorrectly asserts that Narayanaswami discloses a system and 
method similar to that of RFT, wherein query input (feedback) is received 
via speech recognition as claimed. The Appellant respectfully disagrees. 
Specifically, the Office references Col. 8, Lines 22-39, for disclosure of 

feedback, which are excerpted as follows: 

Referring now to FIG. 2, a block diagram of a system for 
searching digital images in an image archive having digital 
images with a plurality of recorded parameters in accordance 
with an embodiment of the present invention is shown. The 
image retrieval system 200 includes a user input/display 202 
for inputting a query and displaying the results of such query 
in accordance with the present invention. It is to be 
understood that the user input/display 202 may be 
conventional devices such as a computer monitor, keyboard 
and mouse (or any equivalent devices). Alternatively, the 
input/display unit 202 may be a liquid crystal display (LCD) 
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touch screen display (or any equivalent user interface). 
Furthermore, the input/display unit 202 may include a 
microphone (not shown) for inputting voice queries. The 
voice queries are processed by a speech processor 204 using 
any conventional speech recognition engine such as the 
commercially available IBM VIA VOICE GOLD engine 
noted above. Narayanaswami, Col 8, Lines 22-39, 

As shown in the above excerpt, Narayanaswami merely describes input of a 
voice query. 

Narayanaswami describes the recordation of parameters with an 

image to later locate the image, as shown in the following excerpts: 

In addition, the parameters to be recorded with each image 
may be specified (or precluded) via voice activated 
commands (e.g., by stating into the microphone 138 "I want 
to have shutter speed recorded with the image"). Such voice 
commands are then received by the CPU 102 via the A/D 
converter 136 and processed in the speech processor module 
104. It is to be appreciated that the digital images may be 
annotated through voice data. For instance, the name of a 
person being photographed can be recorded with the digital 
image by stating "the person being photographed is John 
Jones." It is to be understood that any conventional command 
and control engine for speech recognition may be employed 
in the present invention such as the commercially available 
large vocabulary IBM VIAVOICE GOLD system to perform 
the speech recognition functions in accordance with the 
present invention. Narayanaswami Col 7, Lines 1-16 
(emphasis added). 

Referring now to FIG. 3, a method for searching digital 
images in an image archive in accordance with the present 
invention is shown. To begin, a querying user will input a 
desired query into the system 200 via the input/display unit 
202 (step 300). As discussed above, the query may be input 
via the keyboard or verbally (via the microphone and then 
interpreted by the speech processor module 204 ). The system 
200 will make a determination of what type of query was 
designated based on the query input by the querying user. For 
instance, if it is determined that the desired query is a 
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parameter query (affirmative result in step 302 ) (e.g., the 
query designates certain parameter to be searched), the 
parameter query module 206 will process the query. 
Specifically, the parameter query module 206 will search the 
image database 216 (step 304) and retrieve all imaRcs having 
the parameters designated by the query (step 306) . The 
retrieved images will then be displayed (step 308), 
Narayanaswami, CoL 10, Line 62 to CoL 11, Line 11, 

Thus, although Narayanaswami discloses that the query may be used to 
locate an image, the locating is based on the parameters that are recorded 
with the image. 

3. Claims 53 and 54 are Nonobvious over RFT in view of 
Narayanaswami 

Obviousness cannot be established by combining the teaching of the 
prior art to produce the claimed invention, absent some teaching or 
suggestion supporting the combination. Under section 103, teachings of 
references can be combined only if there is some suggestion or incentive to 
do so. ACSHosp, Sys„ Inc, v. Montefiore Hosp,, 732 F,2d 1572, 221 USPQ 
929 (Fed, Cir, 1984), Thus, the Office may not use the patent application 
as a basis for the motivation to combine or modify the prior art to arrive at 
the claimed invention. 

It is respectfully submitted that the Office has engaged in 
impermissible hindsight reconstruction. RFT does not disclose, teach, or 
suggest speech recognition. Although Narayanaswami describes the 
recordation of speech, Narayanaswami locates images based on parameters 
that were recorded with the image. RFT, however, asserts that the use of 
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parameters for locating images is not desirable, as shown in the following 
excerpt: 

Keyword annotation is the traditional image retrieval 
paradigm. In this approach, the images are first annotated 
manually by keywords. They can then be retrieved by their 
corresponding annotations. However, there are three main 
difficulties with this approach, i.e. the large amount of 
manual effort required in developing the annotations, the 
differences in interpretation of image contents, and 
inconsistency of the keyword assignments among different 
indexers [1, 2, 3]. As the size of image repositories increases, 
keyword annotation approach becomes infeasible. RFT, Page 
L 

Therefore, RFT describes content-based image retrieval which teaches 
away from the use of parameters as asserted by Narayanaswami. As 
previously stated, "[a] prior art reference must be considered in its entirety, 
i.e., as a whole , including portions that would lead away from the claimed 
invention." M,P,E.P, 214L02, citing W,L, Gore & Associates, Inc. v. 
Garlock Inc., 721 K2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, 
denied, 469 U.S. 851 (1984), Therefore, Narayanaswami is not properly 
combinable with RFT because RFT teaches away from the desirability of 
keyword annotation as disclosed by Narayanaswami. 

The suggestion to modify as put forth in the Office Action appears to 
employ an improper "obvious to try" rationale, as is discussed below in 
more detail with reference to MPEP §2145(X)(B), This MPEP section 
states that: 

The admonition that 'obvious to try' is not the standard under 
§103 has been directed mainly at two kinds of error. In some 
cases, what would have been 'obvious to try' would have been 
to vary all parameters or try each of numerous possible 
choices until one possibly arrived at a successful result, where 
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the prior art gave either no indication of which parameters 
were critical or no direction as to which of many possible 
choices is likely to be successful .... In others, what was 
'obvious to try* was to explore a new technology or general 
approach that seemed to be a promising field of 
experimentation, where the prior art gave only general 
guidance as to the particular form of the claimed invention or 
how to achieve it. In re OTarrell 853 K2d 894, 903, 7 
USPQld 1673, 1681 (Fed. Cir. 1988) (citations omitted) 
(emphasis added). 

No guidance has been identified within the references to determine which 
elements to pick or choose from the reference, or of how to couple them to 
somehow arrive at subject matter such as is claimed (see also discussion of 
MPEP §2143 infi-a). Accordingly, neither RFT nor Narayanaswami, alone 
or in combination, disclose, teach or suggest receiving user feedback via 
speech recognition regarding relevancy of one or more of the potentially 
relevant images as claimed in Claims 52 and 54. 

Claim 54 is a dependent claim that depends on Claim 19. The 
foregoing arguments presented Fourth Ground of Rejection, above, in 
reference to the rejection of Claim 19 are hereby incorporated and directed 
to the rejection of Claim 54. Because Claim 54 incorporates all the 
limitations of Claim 19, this claim is also patentable for at least the same 
reasons for which Claim 19 is patentable. 

Accordingly, Claims 52 and 54 satisfy the requirements of 35 U.S.C. 
§ 103(a) so as to be patentable over RFT in view of Narayanaswami. The 
Appellant respectfully requests the Board to overtum the Eighth Ground of 
Rejection. 
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Conclusion 

The Office's basis and supporting rationale for the 35 U.S.C. §102 and 
§103 rejections is not supported by relevant authority, is inconsistent with 
established examination practice and/or does not comport with the teachings of the 
cited references. Appellant respectfully requests that the rejections be overtumed 
and that pending claims 2-8, 10-30, 33, 34, 46-49, and 52-66 be allowed to issue. 



Respectfully Submitted, 



Dated: July 1,2004 




Lee & Hayes, PLLC 
Reg. No. 45,313 
(509) 324-9256 ext. 249 
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APPENDIX OF APPEALED CLAIMS 



1. (Cancelled). 

2, (Previously presented) One or more computer readable media having 
stored thereon a plurality of instructions that, when executed by one or more 
processors, causes the one or more processors to perform acts including: 

receiving an initial image selection; 

generating a plurality of query vectors by extracting, for each query vector, 
one of a plurality of low-level features from the initial image selection; 

selecting a set of potentially relevant images based at least in part on 
distances between the plurality of query vectors and a plurality of feature vectors 
corresponding to low-level features of a plurality of images; 

receiving feedback regarding the relevance of one or more images of the set 
of potentially relevant images; 

generating a new plurality of query vectors based at least in part on the 
feedback; 

generating a weighting of feature elements based at least in part on the 
feedback; and 

selecting a new set of potentially relevant images based at least in part on 
both the weighting of feature elements and distances between the new plurality of 
query vectors and the plurality of feature vectors, wherein the selecting a new set 
of potentially relevant images comprises using a matrix in determining the 
distance between one of the new plurality of query vectors and one of the plurality 
of feature vectors, and further comprising dynamically selecting the matrix based 
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on both a number of images in the set of potentially relevant images for which 
relevance feedback was input and a number of feature elements in the one feature 
vector. 

3. (Original) One or more computer readable media as recited in claim 2, 
wherein the dynamically selecting comprises using a diagonal matrix if the 
number of images in the set of potentially relevant images for which relevance 
feedback was input is less than the number of feature elements in the one feature 
vector, and otherwise using a full matrix. 

4. (Original) One or more computer readable media as recited in claim 2, 
wherein the dynamically selecting comprises: 

if the number of images in the set of potentially relevant images for which 
relevance feedback was input is not less than the number of feature elements in the 
one feature vector, then using one matrix that transforms the query vector and the 
one feature vector to a higher-level feature space and then using another matrix 
that assigns a weight to each element of the transformed query vector and the 
transformed feature vector; and 

if the number of images in the set of potentially relevant images is less than 
the number of feature elements in the one feature vector, then using a matrix that 
assigns a weight to each element of the query vector and the one feature vector. 
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5. (Previously presented) One or more computer readable media as recited 
in claim 2, wherein X represents an image matrix that is generated by stacking 
feature vectors, each of length K, corresponding to the set of potentially relevant 
images for which relevance feedback was received and resulting in an x 
matrix, C represents a weighted covariance matrix of X , det( C ) represents the 
matrix determinant of C, and the matrix comprises a full matrix (PT*) that is 
generated as follows: 

fr* = (det(C))^C-^ 

6. (Previously presented) One or more computer readable media as recited 
in claim 2, wherein represents the kl^^ element of matrix W ^ represents the 

th 

k feature element, represents the standard deviation of the sequence of 's, 
the matrix comprises a diagonal matrix with each diagonal element (w^) being 
generated as follows: 

1 

7. (Previously presented) One or more computer readable media having 
stored thereon a plurality of instructions that, when executed by one or more 
processors, causes the one or more processors to perform acts including: 

receiving an initial image selection; 

generating a plurality of query vectors by extracting, for each query vector, 
one of a plurality of low-level features from the initial image selection; 
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selecting a set of potentially relevant images based at least in part on 
distances between the plurality of query vectors and a plurality of feature vectors 
corresponding to low-level features of a plurality of images; 

receiving feedback regarding the relevance of one or more images of the set 
of potentially relevant images; 

generating a new plurality of query vectors based at least in part on the 
feedback, wherein N represents the number of images in the set of potentially 
relevant images for which relevance feedback has been received, 7r„ represents the 
relevance of image n in the set of images, tt represents a transposition of a vector 
generated by concatenating the individual ;r„ values, and X represents an image 
matrix that is generated by stacking N training vectors corresponding to the set of 
potentially relevant images into a matrix, and wherein each new query vector (q) 
of the new plurality of query vectors is generated as follows: 

^ K X 
<1=^ ; 

generating a weighting of feature elements based at least in part on the 
feedback; and 

selecting a new set of potentially relevant images based at least in part on 
both the weighting of feature elements and distances between the new plurality of 
query vectors and the plurality of feature vectors. 
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8. (Previously presented) One or more computer readable media having 
stored thereon a plurality of instructions that, when executed by one or more 
processors, causes the one or more processors to perform acts including: 

receiving an initial image selection; 

generating a plurality of query vectors by extracting, for each query vector, 
one of a plurality of low-level features from the initial image selection; 

selecting a set of potentially relevant images based at least in part on 
distances between the plurality of query vectors and a plurality of feature vectors 
corresponding to low-level features of a plurality of images; 

receiving feedback regarding the relevance of one or more images of the set 
of potentially relevant images; 

generating a new plurality of query vectors based at least in part on the 
feedback; 

generating a weighting of feature elements based at least in part on the 
feedback; and 

selecting a new set of potentially relevant images based at least in part on 
both the weighting of feature elements and distances between the new plurality of 
query vectors and the plurality of feature vectors, wherein represents a 
summation, over the images in the set of potentially relevant images, of a product 
of a relevance of the image and a distance between the query vector and the 
feature vector, and wherein the selecting a new set of potentially relevant images 
comprises combining, for each image, a weighted distance between the plurality of 
query vectors and the plurality of feature vectors, and wherein the weight (w.) for 
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each of a plurality ( / ) of distances between a query vector and a corresponding 
feature vector is calculated as: 



9. (Cancelled). 

10. (Previously presented) One or more computer readable media having 
stored thereon a plurality of instructions that, when executed by one or more 
processors, causes the one or more processors to perform acts including: 

receiving an initial image selection; 

generating a plurality of query vectors by extracting, for each query vector, 
one of a plurality of low-level features from the initial image selection, wherein 
the low-level features include: a color moments feature, a wavelet based texture 
feature, and a water- fill edge feature; 

selecting a set of potentially relevant images based at least in part on 
distances between the plurality of query vectors and a plurality of feature vectors 
corresponding to low-level features of a plurality of images; 

receiving feedback regarding the relevance of one or more images of the set 
of potentially relevant images; 

generating a new plurality of query vectors based at least in part on the 
feedback; 

generating a weighting of feature elements based at least in part on the 
feedback; and 
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selecting a new set of potentially relevant images based at least in part on 
both the weighting of feature elements and distances between the new plurality of 
query vectors and the plurality of feature vectors. 

1 1 . (Original) A method of selecting between two types of matrixes to be 
used to weight, based on relevance feedback, a plurality of feature elements for 
image retrieval, the method comprising: 

selecting one of the two types of matrixes based on both a number of 
previously retrieved relevant images and a length of a feature vector including the 
plurality of feature elements. 

12. (Original) A method as recited in claim 11, wherein the selecting 
comprises selecting one of the two types of matrixes based on both a number of 
previously retrieved potentially relevant images which were identified by a user as 
being relevant, and the length of the feature vector including the plurality of 
feature elements. 

13. (Original) A method as recited in claim 11, wherein the plurality of 
feature elements are all elements of the same feature. 

14. (Original) A method as recited in claim 11, wherein the selecting 
comprises using a first type of matrix if the number of retrieved relevant images is 
less than the length of the feature vector, and otherwise using a second type of 
matrix. 
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15. (Original) A method as recited in claim 14, wherein the first type of 
matrix comprises a diagonal matrix and wherein the second type of matrix 
comprises a full matrix. 

16. (Original) A method as recited in claim 11, wherein the selecting 
comprises using a first type of matrix if the length of the feature vector exceeds 
the number of retrieved relevant images by at least a threshold amount, and 
otherwise using a second type of matrix, 

17. (Original) A method as recited in claim 16, wherein the first type of 
matrix comprises a full matrix and the second type of matrix comprises a diagonal 
matrix. 

18. (Original) One or more computer readable media including a computer 
program that is executable by a processor to perform the method recited in claim 
11. 

19. (Previously presented) One or more computer readable media having 
stored thereon a plurality of instructions that, when executed by one or more 
processors, causes the one or more processors to perform acts including: 

comparing, for each of a plurality of images, a plurality of feature elements 
from a query vector to a plurality of feature elements from a feature vector 
corresponding to the image; 
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identifying a number of potentially relevant images based on the 
comparing; 

receiving user feedback regarding relevancy of one or more of the 
potentially relevant images; 

re-comparing, for each of the plurality of images, the plurality of feature 
elements from the query vector to the plurality of feature elements from the 
feature vector, including using a matrix to compare the feature elements and 
dynamically selecting a type of matrix to use based on both the user feedback and 
the number of the plurality of feature elements; 

identifying a new set of potentially relevant images based on the re- 
comparing; and 

presenting the new set of potentially relevant images to the user. 

20. (Original) One or more computer readable media as recited in claim 
19, wherein the re-comparing comprises dynamically selecting the type of matrix 
to use based on both a number of the potentially relevant images for which user 
feedback has been received and the number of the plurality of feature elements. 

21. (Original) One or more computer readable media as recited in claim 
19, wherein the dynamically weighting comprises using a first type of matrix if the 
number of retrieved relevant images is less than the length of the feature vector, 
and otherwise using a second type of matrix. 
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22. (Original) One or more computer readable media as recited in 21, 
wherein the first type of matrix comprises a diagonal matrix and the second type 
of matrix comprises a full matrix. 

23. (Original) A method comprising: 

generating a query vector corresponding to a feature of one image; 
identifying a feature vector corresponding to the feature of another image; 
identifying a number of training samples for which relevance feedback has 
been received; 

if the number of training samples either equals or exceeds a threshold 
amount, then determining a distance between the query vector and the feature 
vector including transforming the query vector and the feature vector to a higher- 
level feature space and then assigning a weight to each element of the transformed 
query vector and the transformed feature vector; and 

if the number of training samples does not exceed the threshold amount, 
then determining the distance between the query vector and the feature vector 
including assigning a weight to each element of the query vector and the feature 
vector. 

24. (Original) A method as recited in claim 23, wherein the feature vector 
includes a plurality of feature elements and wherein the threshold amount 
comprises the number of feature elements in the feature vector. 
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25. (Original) A method as recited in claim 23, wherein if the number of 
training samples either equals or exceeds the threshold amount, then determining 
the distance (g), where P is a mapping matrix, q is the query vector, x is the 
feature vector, and A is a weighting matrix, as: 

g = (P(q-x)yA(P(g-x))^ 

26. (Original) A method as recited in claim 23, wherein if the number of 

training samples does not exceed the threshold amount, then determining the 

_> — > 
distance (g), where q is the query vector, x is the feature vector, and A is a 

weighting matrix, as: 

g = (q-xyA(q-x). 

27. (Previously presented) A method as recited in claim 23, further 
comprising: 

repeating the generating, identifying of the feature vector, identifying of the 
number of training samples, and the determining for each of a plurality of features; 
and 

identifying how closely the image and the another image match each other 
by combining the distances between the query vectors and the feature vectors for 
the plurality of features. 

28. (Previously presented) A method as recited in claim 27, wherein the 
identifying how closely the image and the another image match each other 
comprises calculating a weighted summation of each of the individual distances 
for each of the plurality of features. 
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29. (Original) One or more computer readable media including a computer 
program that is executable by a processor to perform the method recited in claim 
23. 

30. (Original) A system comprising: 

a query vector generator to generate a query vector corresponding to a 

feature of one image; 

a comparator, coupled to the query vector generator, to, 

identify a feature vector corresponding to the feature of another image, 

identify a number of training samples for which relevance feedback has 

been received, 

if the number of training samples either equals or exceeds a threshold 
amount, then to determine a distance between the query vector and the feature 
vector including transforming the query vector and the feature vector to a higher- 
level feature space and then assigning a weight to each element of the transformed 
query vector and the transformed feature vector, and 

if the number of training samples does not exceed the threshold amount, 
then to determine the distance between the query vector and the feature vector 
including assigning a weight to each element of the query vector and the feature 
vector. 

31. (Cancelled). 
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32. (Previously presented) A method comprising: 

for one of a plurality of images and each of a plurality of features, 
generating, based on a set of search criteria, a query vector for the feature, 
identifying a feature vector, corresponding to the image, for the feature, and 
determining how closely the feature vector matches the query vector; and 
determining how closely the image matches the set of search criteria based 
on how closely, for the plurality of features, the feature vectors match the query 
vectors, wherein generating the query vector comprises generating the query 
vector based at least in part on user relevance feedback regarding how relevant 
images previously displayed to a user were. 

33. (Previously presented) A method comprising: 

for one of a plurality of images and each of a plurality of features, 
generating, based on a set of search criteria, a query vector for the feature, 
identifying a feature vector, corresponding to the image, for the feature, 
wherein identifying the feature vector includes: 

identifying a low-level feature vector corresponding to the feature; 

and 

mapping the low-level feature vector to a higher level feature space; 
determining how closely the feature vector matches the query vector; and 
determining how closely the image matches the set of search criteria based 
on how closely, for the plurality of features, the feature vectors match the query 
vectors, wherein generating the query vector comprises generating the query 
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vector based at least in part on user relevance feedback regarding how relevant 
images previously displayed to a user were. 

34. (Original) A method as recited in claim 33, wherein the identifying the 
feature vector further comprises incorporating, into the mapping, relevance 
feedback. 

Claims 35-45 (Cancelled). 

46. (Original) A method of generating a query vector to compare to a 
feature vector of another image, the method comprising: 

receiving feedback regarding the relevance of each image of a set of 
images; 

wherein N represents the number of images in the set of images for which 
user relevance feedback has been received, tt^ represents the relevance of image n 
in the set of images, k represents a transposition of a vector generated by 
concatenating the individual tv^ values, and X represents an image matrix that is 
generated by stacking training vectors corresponding to the set of images into a 
matrix; and 

generating a query vector {q) corresponding to one of a plurality of 
features as follows: 

^ K X 
1=ni • 
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47. (Original) One or more computer readable media including a computer 
program that is executable by a processor to perform the method recited in claim 



48. (Original) A method of generating a weight to apply to distances 
between query vectors and feature vectors when combining the distances, the 
method comprising: 

receiving feedback regarding the relevance of each image of a set of 
images; 

wherein represents a summation, over the images in the set of images, of 
a product of a relevance of the image and a distance between the query vector and 
the feature vector; and 

generating a weight ( u- ) for each of a plurality ( / ) of distances between a 
query vector corresponding to one of a plurality ( / ) of features and a feature 
vector corresponding to the one of the plurality ( / ) of features as: 



49. (Original) One or more computer readable media including a computer 
program that is executable by a processor to perform the method recited in claim 
48. 

50. (Cancelled). 

51. (Cancelled). 



46. 
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52. (Previously presented) One or more computer readable media having 
stored thereon a plurality of instructions that, when executed by one or more 
processors, causes the one or more processors to perform acts including: 

receiving an initial image selection; 

generating a plurality of query vectors by extracting, for each query vector, 
one of a plurality of low-level features from the initial image selection; 

selecting a set of potentially relevant images based at least in part on 
distances between the plurality of query vectors and a plurality of feature vectors 
corresponding to low-level features of a plurality of images; 

receiving feedback regarding the relevance of one or more images of the set 
of potentially relevant images, wherein the receiving feedback includes receiving 
feedback via speech recognition; and 

generating a new plurality of query vectors based at least in part on the 
feedback. 

53. (Original) One or more computer readable media as recited in claim 
19, wherein the receiving user feedback regarding relevancy comprises receiving 
user feedback in a range including at least Highly Relevant, Relevant, No Opinion, 
Irrelevant, and Highly Irrelevant. 

54. (Original) One or more computer readable media as recited in claim 
19, wherein the receiving user feedback comprises receiving user feedback via 
speech recognition. 
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55. (Previously presented) One or more computer readable media 
including a computer program that is executable by a processor to cause the 
processor to perform acts of: 

receiving user feedback regarding the relevance of each image of a set of 
images, the user feedback forming a range including at least Highly Relevant, 
Relevant, No Opinion, Irrelevant, and Highly Irrelevant; 

wherein N represents the number of images in the set of images for which user 
feedback has been received, n:^ represents the relevance of image n in the set of 
images, represents a transposition of a vector generated by concatenating the 
individual tt^ values, and X represents an image matrix that is generated by 
stacking N training vectors corresponding to the set of images into a matrix; and 
generating a query vector (q) corresponding to one of a plurality of features as 
follows: 

• 

n=\ 

56. (Original) One or more computer readable media as recited in claim 
55, wherein the receiving user feedback comprises receiving user feedback via 
speech recognition. 

57. (Previously presented) One or more computer readable media having 
stored thereon a plurality of instructions that, when executed by one or more 
processors, causes the, one or more processors to: 
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select a set of potentially relevant images based at least in part on distances 
between a plurality of query vectors extracted from an initial image selection and a 
plurality of feature vectors corresponding to low-level features of a plurality of 
images; 

receive feedback regarding the relevance of one or more images of the set 
of potentially relevant images; 

generate a new plurality of query vectors based at least in part on the 
feedback; 

generate a weighting of feature elements based at least in part on the 
feedback; and 

select a new set of potentially relevant images based at least in part on both 
the weighting of feature elements and distances between the new plurality of query 
vectors and the plurality of feature vectors, wherein represents a summation, 
over the images in the set of potentially relevant images, of a product of a 
relevance of the image and a distance between the query vector and the feature 
vector, and wherein the selecting a new set of potentially relevant images 
comprises combining, for each image, a weighted distance between the plurality of 
query vectors and the plurality of feature vectors, and wherein the weight (u.) for 
each of a plurality ( / ) of distances between a query vector and a corresponding 
feature vector is calculated as: 
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58. (Previously presented) One or more computer readable media as 
recited in claim 57, wherein the plurality of instructions to cause the one or more 
processors to select comprises instructions to cause the one or more processors to 
use a matrix in determining the distance between one of the new plurality of query 
vectors and one of the plurality of feature vectors, and further comprises 
instructions to cause the one or more processors to dynamically select the matrix 
based on both a number of images in the set of potentially relevant images for 
which relevance feedback was input and a number of feature elements in the one 
feature vector, wherein the instructions to dynamically select comprise instructions 
to cause the one or more processors to use a diagonal matrix when the number of 
images in the set of potentially relevant images for which relevance feedback was 
input is less than a number of feature elements in the one feature vector, and 
otherwise using a full matrix. 

59. (Previously presented) One or more computer readable media as 
recited in claim 57, wherein the plurality of instructions to cause the one or more 
processors to select comprises instructions to cause the one or more processors to: 

use one matrix that transforms the query vector and the one feature vector 
to a higher-level feature space and then using another matrix that assigns a weight 
to each element of the transformed query vector and the transformed feature vector 
when the number of images in the set of potentially relevant images for which 
relevance feedback was input is not less than the number of feature elements in the 
one feature vector; and 
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use a matrix that assigns a weight to each element of the query vector and 
the one feature vector when the number of images in the set of potentially relevant 
images is less than the number of feature elements in the one feature vector. 

60. (Previously presented) One or more computer readable media 
including computer readable instructions executable by one or more processors to 
cause the one or more processors to select between two types of matrixes to be 
used to weight, based on relevance feedback, a plurality of feature elements for 
image retrieval, wherein the instructions are further executable to cause the one or 
more processors to select one of the two types of matrixes based on both a number 
of previously retrieved relevant images and a length of a feature vector including 
the plurality of feature elements. 

61. (Previously presented) One or more computer readable media as 
recited in claim 60, wherein instructions executable to cause the one or more 
processors to select comprise instructions executable to cause the one or more 
processors to select one of the two types of matrixes based on both a number of 
previously retrieved potentially relevant images which were identified by a user as 
being relevant, and the length of the feature vector including the plurality of 
feature elements. 

62. (Previously presented) One or more computer readable media as 
recited in claim 60, wherein the plurality of feature elements are all elements of 
the same feature. 
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63. (Previously presented) One or more computer readable media as 
recited in claim 60, wherein instructions executable to cause the one or more 
processors to select comprise instructions executable to cause the one or more 
processors to use a first type of matrix when the number of retrieved relevant 
images is less than the length of the feature vector, and otherwise use a full matrix. 

64. (Previously presented) One or more computer readable media as 
recited in claim 60, wherein instructions executable to cause the one or more 
processors to select comprise instructions executable to cause the one or more 
processors to use a diagonal matrix when the number of retrieved relevant images 
is less than the length of the feature vector, and otherwise use another type of 
matrix. 

65. (Previously presented) One or more computer readable media as 
recited in claim 60, wherein instructions executable to cause the one or more 
processors to select comprise instructions executable to cause the one or more 
processors to use a full matrix when the length of the feature vector exceeds the 
number of retrieved relevant images by at least a threshold amount, and otherwise 
use another type of matrix. 

66. (Previously presented) One or more computer readable media as 
recited in claim 60, wherein instructions executable to cause the one or more 
processors to select comprise instructions executable to cause the one or more 
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processors to use a full matrix when the length of the feature vector exceeds the 
number of retrieved relevant images by at least a threshold amount, and otherwise 
use a diagonal matrix. 



END OF DOCUMENT 
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